perf: show the most frequent text index values, in batches of 10 keys - #3617

Summary

Map key=value text index value lookups now run in parallel batches of 10 keys. Indexes with large dictionary blocks read each key separately and return its most frequent values with topKWeighted, so a high-cardinality key stops early and delays only its own batch. Indexes with small blocks keep main's query shape within each batch. Plain-column text indexes return the most frequent values with topKWeighted, in batches of 10 columns.

Stacked on #3635, which limits these reads to a configurable set of parts (20 median-sized by default).

The problem

getMapTextIndexKeyValues (Map key=value indexes, e.g. idx_log_attr_items) and getTextIndexKeyValues (text indexes on plain columns, e.g. TraceId):

1. Kept the alphabetically first values (groupUniqArray(limit) over a sorted dictionary), not the common ones.
2. Ran up to 100 keys in one query. A key with a value per row (request IDs, UUIDs) has a token per row, and every other key in that query waited on it.
3. Capped the whole query at max_rows_to_read (3M by default) with read_overflow_mode = 'break'. The dictionary is read in sorted token order, so once a high-cardinality key used up the budget, every key that sorts after it came back with no values and no error. On a real LogAttributes lookup, only 22 of 100 keys got values.

The fix

• topKWeighted(limit)(value, cardinality). cardinality is the token's row count in the part, so the values are the most frequent, most frequent first.
• Plain-column text indexes run in queries of at most 10 columns (TEXT_INDEX_KEYS_PER_QUERY). They stream in as each query finishes.
• Query shape depends on the index's dictionary block size, for Map key=value indexes:
  • 4096 or more: queries of at most 10 keys, each key in its own UNION ALL branch reading at most 20,000 tokens (TEXT_INDEX_TOKENS_PER_KEY), so a high-cardinality key's read stops early. No max_rows_to_read cap ('0', after the team's settings): the per-key LIMIT bounds each read.
  • Smaller, including ClickHouse's default of 512: main's query shape, in queries of at most 10 keys: one read with OR'd startsWith filters, groupUniqArray(limit), and the max_rows_to_read cap with read_overflow_mode = 'break'. It keeps main's known limits: the alphabetically first values, and keys that sort after a high-cardinality key can come back empty. Batches of 10 limit that to at most 9 other keys. The cap is marked with a shortcut: comment.
  • The block size comes from dictionary_block_size in the index definition, then the table's text_index_dictionary_block_size, then the server's MergeTree setting, then 512.
  • Plain-column text indexes keep one branch per column. Each branch reads a different index, so nothing is read twice. No max_rows_to_read cap either.

Why the block size decides

Every read of mergeTreeTextIndex() opens each part's index and loads its sparse index, which has one entry per dictionary block. Separate UNION ALL branches don't share that work.

On a local benchmark (1M rows, 3M tokens, 10 parts), a batch of 10 keys:

The sparse index was 17.7 KB per part with 512-token blocks and 1.3 KB with 8192. On ClickHouse Cloud, an earlier version with per-key branches on default-block indexes measured about 0.18 CPU-s and 10–30 MiB of cache reads per part per branch (26-part otel_logs: 804 ms for one key, 772 ms for a key that doesn't exist). With 8192-token blocks that fixed cost is small enough for per-key branches to pay off.

Cost and known limits

• Batches of 10 mean up to 10× more queries than main. Each opens every part's index once (small blocks) or once per key (large blocks).
• On small-block indexes, a high-cardinality key reads all its tokens in the parts read, so its batch is slower. Results stay complete.
• Block size detection is per table setting. Parts written before someone changed a table's text_index_dictionary_block_size keep their old block size. This only picks the query shape; results are the same either way.
• topKWeighted uses about 1.6× the CPU per token of groupUniqArray. Memory is the same.
• getEngineSetting moved from DBRowTable.tsx to common-utils/core/utils.ts so both use one parser for engine_full settings.

How to reproduce

Starting point: yarn dev against ClickHouse 26.3+, with the OTel otel_logs schema (key=value text indexes idx_log_attr_items, idx_res_attr_items), and a Map key with one rare value that sorts first and one common value, e.g. map('freq.key', if(number = 0, 'a-rare', 'z-common')) from numbers(4).

On main:

1. On Search, turn on showing all values in the filter panel. freq.key lists a-rare first, and system.query_log shows one values query for all keys.

On this branch:

1. Repeat step 1 with the seed schema (dictionary_block_size = 8192). freq.key lists z-common first.
2. In system.query_log: with the seed schema's dictionary_block_size = 8192, there is one values query per 10 keys, with one UNION ALL branch per key. Recreate the index without dictionary_block_size (default 512) and each values query has main's shape for 10 keys.

References

• Related PRs: perf: share one Map key read between concurrent callers #3611, perf: read a configurable set of parts for text index key and value lookups #3635. Absorbs the row cap change from fix: drop the row cap from text index value lookups and read a limited set of parts #3637 (closed).